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CLAIM CHART 



Claim 1 


Comments 


A machine-based method comprising 


In the prior approach, the model developers/users 
used their knowledge, experience and expertise to 
develop individual custom-designed models 
manually using conventional statistical software 
package, in particular, software provided by SAS 
Institute Inc., North Carolina 


providing a graphical user interface that 
enables a user of a model generation tool to 
view and manage subgroups of 
contributory and non-contributory variables 
associated with generation of a predictive 
model including source variables 
associated with attributes of the original 
data and derived variables subsequently 
modified from the source variables. 


In the prior approach, there was no graphical user 
interface that enabled a user of a model generation 
tool to view and manage subgroups of 
"contributory and non-contributory variables 
associated with generation of a predictive model 
including source variables associated with 
attributes of the original data and derived variables 
subsequently modified from the source variables." 


Claim 2 




The method of claim 1 in which the 
derived variables include constructed 
variables, and transformed variables. 


In the prior approach, there was no graphical user 
interface that enabled a user of a model generation 
tool to view and manage derived variables which 
included "constructed variables and transformed 
variables." 


Claim 3 
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The method of claim 1 also including 
enabling the user to move variables 
between subgroups using a pointer. 


In the prior approach, there was no interactive 
graphical interface for "[moving] variables 
between subgroups using a pointer." The model 
developers/users used conventional statistical 
packages having manually entered interpretive 
code to assign variables to subgroups. 


Claim 4 




The method of claim 1 in which the user 
interface enables a user to view, with 
respect to a selected variable, its definition 
and its response distribution relative to an 
outcome variable. 


In the prior approach, there was no graphical user 
interface for viewing "definition and response 
distribution" of a selected variable. The model 
developers/users used conventional statistical 
packages to manually enter interpretive code that 
could generat print-outs showing variable 
definition and response distribution relative to the 
outcome variable. 


ClaimS 




The method of claim 1 in which only 
variables within one of the subgroups are 
used in the model that is generated. 


In the prior approach, there was no graphical 
interface for segregating variables from one of the 
subgroups so that "only variables within [the 
subgroup] is used in the model that is generated." 
The model developers/users used conventional 
statistical packages to manually enter interpretive 
code that could select and list variables from one 
of the subgroups. 


Claim 6 




The method of claim 1 in which the 
subgroup to which each variable belongs is 
stored persistently. 


In the prior approach, there was no means of 
graphically selecting a subgroup of variables to be 
stored persistently. The model developers/users 
used conventional statistical packages to manually 
enter interpretive code that only stored model 
results persistently including which variables were 
used in the model. 


Claim 7 




A machine-based method comprising 


In the prior approach, the model developers/users 
used their knowledge, experience and expertise to 
develop individual custom-designed models 
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manually using conventional statistical software 
package, in particular, software provided by SAS 
Institute Inc., North Carolina 


in connection with a project in which a user 
generates a predictive model based on 
historical data about a system being 
modeled, enabling the user through a 
graphical user interface to manage and 
view information about distributions and 
interactions based on strengths of 
measurements of predictor variables 
associated with the data. 


In the prior approach, there was no graphical user 
interface to manage and view "distributions and 
interactions based on strengths of measurements of 
predictor variables associated with the [historical] 
data." The model developers/users used 
conventional statistical packages to manually enter 
interpretive code that merely generated tabular 

r* i • i ii1 1*j • 1 1 

information about the predictor variables. 


Claim 8 




The method of claim 7 in which the 
information includes a status of a predictor 
variable. 


In the prior approach, there was no graphical 
interface that provided information including the 
"status of a predictor variable." The model 
developers/users used conventional statistical 
packages to manually enter interpretive code that 
sought to determine the status of a predictor 
variable. 


Claim 9 




The method of claim 7 in which the 
information includes a status of a class of 
non-predictor variables. 


In the prior approach, there was no graphical user 
interface that provided information including the 
"status of a class of non-predictor variables." The 
model developers/users used conventional 
statistical packages to manually enter interpretive 
code that sought to determine the status of non- 
predictor variables. 


Claim 10 




The method of claim 7 in which the user 
interface enables the user to point and click 
to cause display of information about the 
variable. 


In the prior approach, there was no interactive 

1*1 'if* iii -til 
graphical user interface that enabled a user to 

"point and click to cause display of information 

about [a] variable." 


Claim 11 




The method of claim 10 in which the 
information about the variable comprises at 
least one of: a description, a definition, a 
history of transformations, a response 


in me prior appro acn, mere was no interactive 
graphical user interface showing information about 
the variable that included any of at least "a 
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graph, and a link to the graph. 


description, a definition, a history of 
transformations, a response graph, and a link to the 
graph." 


Claim 12 




The method of claim 1 1 in which 
invocation of the link to the graph causes 
display of the probability of an event as a 
function of the variable. 


In the prior approach, there was no interactive 
graphical user interface having a link that when 
invoked "causes display of the probability of an 
event as a function of [a] variable." 


Claim 13 




The method of claim 7 also including 
enabling the user to cause a variable to be 
transformed, and including the transformed 
version of the variable in the information 
about predictor variables. 


In the prior approach, there was no interactive 
graphical user interface which enabled the analyst 
to transform a variable and then "including the 
transformed version of the variable in information 
about predictor variables." The model 
developers/users used conventional statistical 
packages to manually enter interpretive code that 
transformed a variable and use the variable in a 
model. 


Claim 14 




The method of claim 7 in which the 
predictor variables are grouped in the user 
interface, and also including enabling the 
user to move one or more than one variable 
from one group to another. 


In the prior approach, there was no interactive 
graphical user interface for grouping or for 
"enabling [a] user to move one or more than one 
variable from one group to another." 


Claim 15 




The method of claim 7 in which the 
graphical user interface represents filtered 
views of a data dictionary of an entire set 
of variables of the model. 


m tne prior appro acn, mere was no data dictionary 
object and no graphical user interface that 
[represented] filtered views of a data dictionary. 


Claim 16 




The method of claim 7 in which the user 
interface enables the user to point and click 
to cause display of decision partition trees 
of the collection of predictor variables. 


In the prior approach, there was no interactive 
graphical user interface that enabled model 
developers/users to point and click to cause 
display of decision partition trees of [a] collection 
of predictor variables 


Claim 17 
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The method of claim 7 in which the user 
interface enables the user to point and click 
to cause display of response distribution 
functions of the collection of predictor 
values. 



In the prior approach, there was no interactive 
graphical user interface that enabled model 
developers/users "to point and click to cause 
display of response distribution functions of [a] 
collection of predictor values." The model 
developers/users used conventional statistical 
packages to manually enter interpretive code that 
generated "response distribution functions of the 
collection of predictor values." 
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DECLARATION IN RESPONSE TO REQUEST FOR INFORMATION IN ACTION DATED 

MARCH 29, 2007 

I, Richard Mansfield, declare: 

I am the Chief Architect and Vice President Platform Technologies of Fortelligent, Inc., 
the assignee of this patent application. 

My comments below are each preceded by the related requests of the examiner (in small, 
bold type). 

I) What were the specific details of the modeling system referred to in the admission? 

More than a year prior to April 16, 2004, the assignee's employees used individual 
custom-designed models to generate customer relationship management (CRM) information for 
customers of the assignee. The individual custom-designed models were each generated 
manually using the experience, expertise, and knowledge of model developers and the 
capabilities of commercially available statistical software, in particular, solutions from SAS 
Institute Inc., North Carolina. 

From its records, the assignee is unable to reconstruct the contents of any of the specific 
individual custom-designed models in use prior to April 16, 2003, or the manner in which they 
were generated other than as disclosed in the information disclosure statement of May 27, 2005. 



a) Do you have any documentation, either hard copies or in electronic form, showing 
this modeling system? If yes, please provide documentation in response to this interrogatory. 
Documentation should be interpreted broadly to include, yet not be limited to: press releases, 
advertisements, brochures, product manuals, web pages, patents and patent applications 
(both U.S. and foreign), presentations, technical journals, and product specifications. 
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None to the best of the undersigned's knowledge. 

b) Do any of the parties subject to a duty of disclosure under 37 CFR 1.56 know of 
where such documentation, as recited above, may be found? Please state the location in 
response to this question. 

See the reply to a) above. 

c) Using the language of the claims, what are the specific differences between the 
modeling system that is mentioned in the admission and the claims? 

A claim chart is attached. 

d) Were there any other modeling systems created by applicants? If so, please provide 
specific details of these modeling systems, using the language of the claims, and explain how 
they differ from the claimed invention. 

No other modeling systems were created more than a year before April 16, 2004. 
Beginning in March 2003, the assignee began to develop a software application to assist users in 
developing models. Because no available products met identified requirements, a new computer- 
based modeling system was developed. The architecture and design of the system were complete 
by June 2003. Implementation of the initial version was complete by November 2003. However, 
in August 2003, the model project insight feature illustrated in figure 27D was sufficiently 
complete that customer data was analyzed and the graphical and tabular reports were used in a 
project for a customer. 

e) Were there any other modeling systems created by applicants? If so, please provide 
documentation of these systems or state where such documentation may be found. 

The computer-based modeling system referred to in d) was the sole modeling system 
created as it provided the required features. 

II) Was the modeling system disclosed in the admission the subject of any of the 
following types of transactions: offered for sale, bartered or exchanged, licensed, donated or 
given away, or used for experimental use? Please answer the following interrogatories for 
each transaction. 

a) Where did this transaction occur? 

b) What were the details of the transaction? 

c) What where the details of the system that formed the basis for this transaction? 
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d) When did this transaction occur? 

e) Who were the parties involved in the transaction? 

More than a year prior to April 16, 2004, the assignee received raw customer data from 
customers and transformed the data into customer relationship management (CRM) information 
(predictive scores) for customers of the assignee using the experience, expertise, and knowledge 
of model developers and the capabilities of commercially available model creation software, in 
particular, solutions from SAS Institute Inc., North Carolina. The following table lists the details 
of these transactions: 



Contracting Party 


Modeling Service 
Provided 


Model Method 


Date of Contract 


Cross Country 


Response Models for 
Marketing 


Manually developed by 
skilled model 
developers/analysts 
assisted by standard 
statistical software 


Annual Contract, prior to 
April 2003, with series 
of contract renewals 


Member Works (now 
Vertue) 


Response Models for 
Marketing 


Manually developed by 
skilled model 
developers/analysts 
assisted by standard 
statistical software 


Annual Contract, prior to 
April 2003, with series 
of contract renewals 


Other parties to whom 
pitches were made 


[details no longer 
recalled] 


[details no longer 
recalled] 





III) Was the modeling system of this application or of the admission publicly disclosed? 

No. 



a) Do yon have any documentation, either hard copies or in electronic form, showing 
what was publicly disclosed? If yes, please provide documentation in response to this 
interrogatory. Documentation should be interpreted broadly to include, yet not be limited to: 
press releases, advertisements, brochures, product manuals, web pages, presentations, 
technical journals, and product specifications. 

None other than materials already submitted. 



b) Do any of the parties subject to a duty of disclosure under 37 CFR 1.56 know of 
where such documentation, as recited above, may be found? Please state the location in 
response to this question. 
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The assignee does not know of the location or of any other party that may know of the 
location of any such documentation. 



4. In response to this requirement, please provide the title, citation and copy of each 
publication that any of the applicants relied upon to develop the disclosed subject matter that 
describes the applicant's invention, particularly as to developing "transforming of the 
variables" and "the adjusting of the population of variables". For each publication, please 
provide a concise explanation of the reliance placed on that publication in the development of 
the disclosed subject matter. 

In developing the disclosed subject matter, the applicant relied on portions of the items 
described in its information disclosure statements. With particular regard to "transforming of the 
variables" and "adjusting of the population of variables," the inventors used their knowledge, 
experience and expertise along with such literature disclosed in the attached and previous 
information disclosure statements to develop the recited features. For the examiner's reference, 
biographies of the inventors are attached as exhibits 1 through 3. 

5. In response to this requirement, please provide the title, citation and copy of each 
publication that any of the applicants relied upon to draft the claimed subject matter. 

For each publication, please provide a concise explanation of the reliance placed on that 
publication in distinguishing the claimed subject matter from the prior art 



The inventors are unable to recall any specific such reliance that may have been placed. 

6. In response to this requirement, please provide the names of any products or 
services that have incorporated the claimed subject matter. 

OASYS™ Analytic Platform; this product was not offered for sale prior to April 16, 

2003. 

7. In response to this requirement, please state the specific improvements of the 
subject matter in claims 1 and 7 over the disclosed prior art and indicate the specific 
elements in the claimed subject matter that provide those improvements. For those claims 
expressed as means or steps plus function, please provide the specific page and line numbers 
within the disclosure that describe the claimed structure and acts. 



See the related Response filed by the applicant. 



All statements made in this declaration of my own knowledge are true and all statements 
made on information and belief are believed to be true; and further these statements were made 



10/01/2007 13:29 FAX 



@003/003 
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with the knowledge that willful false statements and the like so made are punishable by fine or 
imprisonment, or both, under Section 1001 of Title 18 of the United States Code and that such 
willful false statements may jeopardize the validity of the application or any patent issued on the 
application. 

Respectfully submitted, 

Date: 
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CLAIM CHART 



Claim 1 




A machine-based method comprising 


In the prior approach, the model developers/users 
used their knowledge, experience and expertise to 
develop individual custom-designed models 
manually using conventional a statistical software 
package, in particular, software provided by SAS 
Institute Inc., North Carolina. 


receiving historical mnlti -dimensional data 

IWwlYlUg lllOl-Vyl I VC11 ill villi U 1111 VI lOlv-ZHClrl UCllCl 

representing multiple variables, 


The nrior anrvroarh used historical multi- 

1 11V L/LLWl C*jjL71 V ClVll UJvU lllkJUVl 1VC11 111 LAI 11 

dimensional data. 


transforming variables into more predictive 
variables, including Bayesian renormalized 
variables, linearly transformed and non- 
linearly transformed variables and imputed 
missing values for categorical or 
continuous variables, 


In the prior approach, the model developers/users 
manually transformed variables into more 
predictive variables using conventional statistical 
tools. 

The prior approach included imputation of missing 
values for categorical or continuous variables but 
only with traditional methods. 

The prior approach did not include Bayesian 
renormalization. 


pruning variables for which the data is 
sparse or missing, 


In the prior approach, pruning of variables for 
which the data was spare or missing was done 
manually. 


adjusting a population of variables to 
represent main effects exhibited by the data 


In the prior approach, mainly first-order interaction 
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and significant interaction and non-linear 
effects exhibited by the data. 


effects were considered. 


nQincr flip rirvrvnlfltinTi nf variflhlpQ tn 

generate a predictive model for interacting 
with a commercial system. 


In the nrior annroach the model devel oners/users 
used their knowledge, experience and expertise to 
manually generate a preuictive muuei lur 
interacting with a commercial system. 


Claim 2 




The method of claim 1 in which adjusting 
the population of variables to represent 
interaction effects includes stages of main 
effect interactions, main effects with main 
effect interactions and excluded variable 
interactions and main effects with main 
effect interactions and excluded variable 

intprapfinnQ tnopth pr with pypIiiHp/1 varifihlp 

Uilvl uVUUXla LUgGUlCd Wllil C-AVlUUtHJ VallaUlv 

combined interactions. 


In the prior approach, mainly first-order interaction 
effects were considered. 


Claim 3 




The method of claim 1 in which the 
predictive model predicts behavior of a 
current customer with respect to retention 
of a current service or product of a vendor. 


In the prior approach, the model developers/users 
manually "[predicted the] behavior of a current 
customer with respect to retention of a current 
service or product of a vendor," using their 
Knowieuge, experience ana expertise aiong wiui 
conventional statistical software packages. 


Claim 4 




The method of claim 1 in which the 

rvrpHirtivp mnrlpl rvrpfliPtQ hphavirvr of a 

current customer with respect to risk of 
asserting claims, loon payment or 
prepayment to a vendor. 


The prior approach did not include this feature. 


ClaimS 




The method of claim 1 in which the 
predictive model predicts behavior of a 
current customer with respect to usage of a 
current service or product of a vendor. 


In the prior approach, the modelers/users manually 
"[predicted the] behavior of a current customer 
with respect to usage of a current service or 
product of a vendor," using their knowledge, 
experience and expertise along with conventional 
statistical software packages. 
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Claim 6 




The method of claim 1 also including 
enabling a user to reconstruct a sequence of 
choices involved in the creation of the 
predictive model. 


The prior approach did not include this feature. 


Claim 7 




A machine-based method comprising 


In the prior approach, the model developers/users 
used their knowledge, experience and expertise to 
manually develop individual custom-designed 
models using conventional statistical software 
package, in particular, software provided by SAS 
Institute Inc., North Carolina. 


in cnnnertinn with a typhi pf*t haseH An 
in wiiiivvuuii wim a uiujvvi i/ciov>Ai vii 

historical data about a system being 
moueiea, generating a preuicuve mouei, 
and 


The prior approach used historical data about a 

svstem The mnHel Hevplrvners/iisers iiseH their 

knowledge, experience and expertise to and a 
commercially avanaoie statisucai sonware 
package to manually generate a predictive model. 


providing to the a user through a graphical 
user interface a sentience of dimension 

14 O \_- 1 111 I VI JLtlv'V ti JwUUvHvv v/JL Vlllll vllOlWli 

reduction having two or more steps. 


The prior approach did not include this feature. 


Claim 8 




The method of claim 7 in which the system 
being modeled comprises behavior of 
prospective or current customers of a 

offered by the vendor. 


This feature of the system being modeled was part 
of the nrior annroach 


Claim 9 




The method of claim 7 in which the 
nredictive model nredicts behavior of a 
prospective or current customer with 
respect to purchase of a product or service 
of a vendor. 


This feature of the predictive model was in the 
prior approach. 


Claim 10 




The method of claim 7 in which the 
predictive model predicts behavior of a 
current customer with respect to retention 


This feature of the predictive model was in the 
prior approach. 
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oi a current service or prouuct or a vendor. 




Claim 11 




The method of claim 7 in which the 
predictive model predicts behavior of a 
current customer with respect to risk of 
asserting claims loan navment or 
prepayment to a vendor. 


The prior approach did not include this feature. 


Claim 12 




The method of claim 7 in which the 
predictive model predicts behavior of a 

niTTfrit fniQtnmfM* with r(*<zx\€ k c\ tn iiQacrf* nf a 

current service or product of a vendor. 


This feature of the predictive model was in the 
prior approach. 


Claim 13 




The method of claim 7 in which the user 
interface controls staging of the sequence 
of model generation activities. 


The prior approach did not include this feature. 



EXHIBIT C 
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J Dr. if&ted Mansflelcl, Chief Architect 
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B«rtellii«nf 



Dr. Mansfield has more than 15 years of successful, progressive experience in all 
phases of mission-critical software product and systems development. He is the author 
or co-author of a number of software patents. His role at Fortelligent is to design and 
develop Forteliigenfs OASYS Analytic Platform™ and to provide technical oversight of 
Forteliigenfs systems, software and processes to ensure optimization for business 
advantage. 

Dr. Mansfield earned his Ph.D. from Harvard University and a Master's from Yale 
University in Mathematics. He was a Visiting Scholar at the Johns Hopkins School of 
Medicine as well. 



EXHIBIT D 



itppii&n ^gWfe Senior Mm WtrnMrnt 
Beftfaf M0fi®g®rmM Biography 



Steve Pinto has been at the forefront of predictive analytics for over 30 years. Over 
that time he has formulated many of the modem-day methodologies used in database 
marketing and modeling. He is responsible for Product Development, Account 
Management, and client deliverables at Fortelligent. His prior experience includes: 

At Mercer Management Consulting, a global strategy firm, Steve was a Vice President 
and a lead developer of the Customer Relationship Management practice. He was 
responsible for developing the worldwide practice for CRM at Mercer. The practice 
develops analytical CRM program on behalf of diversified Global 2,000 clients and 
manages those programs on an ongoing basis to help generate increasing shareholder 
value. 

Prior to his work at Mercer Management Consulting, Steve was the Managing Director 
of the Analytic Consulting Group at Epsilon, a former division of American Express. 
Steve was responsible for creating and deploying analytical CRM programs on behalf 
of the largest financial institutions in the country. 

Before Epsilon, Steve Co-Founded Applied Research of Cambridge, a worldwide 
market research firm later acquired by Cap Gemini, a global strategy consulting firm. 

Steve is a widely published author on the subject of predictive analytics and is on 
several editorial boards including the Journal of Database Marketing, 
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Fortelligent's Scientific Advisor is the world-renown statistician, Professor Donald 
Rubin. His role at Fortelligent has been to contribute to the development of 
Fortelligent's Platform by designing unique algorithms and methodologies that enable 
Fortelligent to maintain an ongoing competitive edge. 

Professor Rubin is a world-leading thinker in Statistical Science. In 2002 he was one of 
the most quoted scholars in the field of statistics. Currently he is Chairman of the 
Statistics Department at Harvard University, a position he has held for 20 years. He is 
the John L. Loeb Professor of Statistics and key PhD advisor to Candidates for degrees 
in Statistics at Harvard as well. 

Prior roles held at Harvard University include: 

• Chairman, Department of Statistics Senior Search Committee 

• Director of Graduate Studies, Department of Statistics 
Faculty Staff, Program on Educational Policy and Governance 

Professor Rubin is a widely known force in the statistical community, having published 
over 300 articles and book on various statistical-related topics. 
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DECLARATION IN RESPONSE TO REQUEST FOR INFORMATION IN ACTION DATED 

MARCH 29, 2007 

I, Richard Mansfield, declare: 

I am the Chief Architect and Vice President Platform Technologies of Fortelligent, Inc., 
the assignee of this patent application, 

My comments below are each preceded by the related requests of the examiner (in small, 
bold type). 

I) What were the specific details of the modeling system referred to in the admission? 

More than a year prior to April 16, 2004, the assignee's employees used individual 
custom-designed models to generate customer relationship management (CRM) information for 
customers of the assignee. The individual custom-designed models were each generated 
manually using the experience, expertise, and knowledge of model developers and the 
capabilities of commercially available statistical software, in particular, solutions from SAS 
Institute Inc., North Carolina. 

From its records, the assignee is unable to reconstruct the contents of any of the specific 
individual custom-designed models in use prior to April 16, 2003, or the manner in which they 
were generated other than as disclosed in the information disclosure statement of May 27, 2005. 



a) Do you have any documentation, either hard copies or in electronic form, showing 
this modeling system? If yes, please provide documentation in response to this interrogatory. 
Documentation should be interpreted broadly to include, yet not be limited to: press releases, 
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advertisements, brochures, product manuals, web pages, patents and patent applications 
(both U.S. and foreign), presentations, technical journals, and product specifications. 

None to the best of the undersigned's knowledge. 

b) Do any of the parties subject to a duty of disclosure under 37 CFR 1.56 know of 
where such documentation, as recited above, may be found? Please state the location in 
response to this question. 

See the reply to a) above. 

c) Using the language of the claims, what are the specific differences between the 
modeling system that is mentioned in the admission and the claims? 

A claim chart is attached. 

d) Were there any other modeling systems created by applicants? If so, please provide 
specific details of these modeling systems, using the language of the claims, and explain how 
they differ from the claimed Invention. 

No other modeling systems were created more than a year before April 16, 2004. 
Beginning in March 2003, the assignee began to develop a software application to assist users in 
developing models. Because no available products met identified requirements, a new computer- 
based modeling system was developed. The architecture and design of the system were complete 
by June 2003. Implementation of the initial version was complete by November 2003. However, 
in August 2003, the model project insight feature illustrated in figure 27D was sufficiently 
complete that customer data was analyzed and the graphical and tabular reports were used in a 
project for a customer. 

e) Were there any other modeling systems created by applicants? If so, please provide 
documentation of these systems or state where such documentation may be found. 

The computer-based modeling system referred to in d) was the sole modeling system 
created as it provided the required features. 



II) Was the modeling system disclosed in the admission the subject of any of the 
following types of transactions: offered for sale, bartered or exchanged, licensed, donated or 
given away, or used for experimental use? Please answer the following interrogatories for 
each transaction. 
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a) Where did this transaction occur? 

b) What were the details of the transaction? 

c) What where the details of the system that formed the basis for this transaction? 

d) When did this transaction occur? 

e) Who were the parties involved in the transaction? 

More than a year prior to April 16, 2004, the assignee received raw customer data from 
customers and transformed the data into customer relationship management (CRM) information 
(predictive scores) for customers of the assignee using the experience, expertise, and knowledge 
of model developers and the capabilities of commercially available model creation software, in 
particular, solutions from SAS Institute Inc., North Carolina. The following table lists the details 
of these transactions: 



Contracting Party 


Modeling Service 
Provided 


Model Method 


Date of Contract 


Cross Country 


Response Models for 
Marketing 


Manually developed by 
skilled model 
developers/analysts 
assisted by standard 
statistical software 


Annual Contract, prior to 
April 2003, with series 
of contract renewals 


Member Works (now 
Vertue) 


Response Models for 
Marketing 


Manually developed by 
skilled model 
developers/analysts 
assisted by standard 
statistical software 


Annual Contract, prior to 
April 2003, with series 
of contract renewals 


Other parties to whom 
pitches were made 


[details no longer 
recalled] 


[details no longer 
recalled] 





III) Was the modeling system of this application or of the admission publicly disclosed? 

No. 

a) Do you have any documentation, either hard copies or in electronic form, showing 
what was publicly disclosed? If yes, please provide documentation in response to this 
interrogatory. Documentation should be interpreted broadly to include, yet not be limited to: 
press releases, advertisements, brochures, product manuals, web pages, presentations, 
technical journals, and product specifications. 



None other than materials already submitted. 
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b) Do any of the parties subject to a duty of disclosure under 37 CFR 1.56 know of 
where such documentation, as recited above, may be found? Please state the location in 
response to this question. 

The assignee does not know of the location or of any other party that may know of the 
location of any such documentation. 



4. In response to this requirement, please provide the title, citation and copy of each 
publication that any of the applicants relied upon to develop the disclosed subject matter that 
describes the applicant's invention, particularly as to developing "transforming of the 
variables" and "the adjusting of the population of variables". For each publication, please 
provide a concise explanation of the reliance placed on that publication in the development of 
the disclosed subject matter. 

In developing the disclosed subject matter, the applicant relied on portions of the items 
described in its information disclosure statements. With particular regard to "transforming of the 
variables" and "adjusting of the population of variables," the inventors used their knowledge, 
experience and expertise along with such literature disclosed in the attached and previous 
information disclosure statements to develop the recited features. For the examiner's reference, 
biographies of the inventors are attached as exhibits 1 through 3. 

5. In response to this requirement, please provide the title, citation and copy of each 
publication that any of the applicants relied upon to draft the claimed subject matter. 
For each publication, please provide a concise explanation of the reliance placed on that 
publication in distinguishing the claimed subject matter from the prior art. 

The inventors are unable to recall any specific such reliance that may have been placed. 

6. In response to this requirement, please provide the names of any products or 
services that have incorporated the claimed subject matter. 

OASYS™ Analytic Platform; this product was not offered for sale prior to April 16, 

2003. 

7. In response to this requirement, please state the specific improvements of the 
subject matter in claims 1 and 7 over the disclosed prior art and indicate the specific 
elements in the claimed subject matter that provide those improvements. For those claims 
expressed as means or steps plus function, please provide the specific page and line numbers 
within the disclosure that describe the claimed structure and acts. 



10/01/2007 13:29 FAX 
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See the related Response filed by the applicant. 

All statements made in this declaration of my own knowledge are true and all statements 
made on information and belief are believed to be true; and further these statements were made 
with the knowledge that willful false statements and the like so made are punishable by fine or 
imprisonment, or both, under Section 1001 of Title 1 8 of the United States Code and that such 
willful false statements may jeopardize the validity of the application or any patent issued on the 
application. 

Respectfully submitted, 

Date: 




EXHIBIT B 
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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

Applicant : Stephen K. Pinto et al. Art Unit : 2121 

Serial No. : 10/826,947 Examiner : Tejal Garni 

Filed : April 16, 2004 Conf.No. : 1609 

Title : PREDICTIVE MODEL VALIDATION 

MAIL STOP AMENDMENT 

Commissioner for Patents 
P.O. Box 1450 
Alexandria, VA 22313-1450 



CLAIM CHART 



Claim 1 


Comments 


A machine-base method comprising 


In the prior approach, the model developers/users 
used their knowledge, experience and expertise to 
develop individual custom-designed models 
manually using conventional statistical software 
package, in particular, software provided by SAS 
Institute Inc., North Carolina 


in connection with a project, generating a 
predictive model based on historical data 
about a system being modeled, enabling 
the user to validate a model development 
process with a predictive model between 
at least two subsets of the historical data, 
and interacting with the system being 
modeled based on the predictive model. 


In the prior approach, the model developers/users 

validate a specific model manually using a 
conventional approach and a conventional statistical 
software package, in particular, software provided 
by SAS Institute Inc., North Carolina, rather than 
"enabling the user to validate a model development 
process." 


Claim 2 




The method of claim 1 in which a user 
interface display project goals enabling 
the user to assess model project 
performance wherein the project goals 
comprise at least one of: cumulative lift 
over an interval of interest, degree of 
monotonicity, or concordance scores. 


In the prior approach, the model developers/users 
used their knowledge, experience and expertise to 
manually develop individual custom-designed 
models using conventional statistical software 
package, in particular, software provided by SAS 
Institute Inc., North Carolina, which could print 
tables for comparing conventional measures such as 
cumulative lift, concordance (minimum false 
negatives or minimum false positives), and 
monotonicity. In the prior approach, there was no 
user interface that displayed "project goals enabling 
the user to assess model project performance . . .". ! 
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Claim 3 




ine rneuiou or ciaim z also inciuumg 
identifying that the model does not 
produce at least a predefined degree of lift 
for at least one of the validation datasets. 


In the prior approach, as one criterion of 

npHYvrma'npp a licpr mannallv nrvnlH iiQf* low lifl" i<5 a 

JJCl Il/IIIiCUlVW, <X UOvl iiiCUXUCtll Y VVUIU Udv IvW JUJ.I id U 

conventional heuristic. 


Claim 4 




The method of claim 3 also including 
enabling a user to choose interactively at 
least one model development criterion 
change or transformation or interaction of 
variables to improve a fit of the model. 


In the prior approach, the model developers/users 
manually transformed variables into more predictive 
variables using conventional statistical tools using 
expert estimation and guesswork in making 
additional transformations and interactions. The 

iicp»r ii/ao tint pnnnlpn 4t tr\ Phnncp intf*t"Qr > ti"v/P k l\/ cit 
UoCi Woo UUl CllaUlCU lu WlUUoC HllCTavU Vviy al 

least one model development criterion or 
transtormation or interaction 01 vanaoies to improve 
the fit of the model." 


ClaimS 




The method of claim 4 also including 
graphically displaying and comparing 
measures of performance for the 
validation dataset and a training dataset. 


The prior approach did not include "graphically 
displaying and comparing measures of performance 

lur U1C VallUallUll UalaoCl aUU LUC LlaillLUg uaiaoci. 

Users could, however, use a third party software 
package, sucn as microsoit nxcei, to construct a 
comparison chart. 


Claim 6 




A machine-based method comprising 


In the prior approach, the model developers/users 
used their knowledge, experience and expertise to 
manually develop muiviauai custom-aesignea 
models using conventional statistical software 
package, in particular, software provided by SAS 
Institute Inc., North Carolina 


in connection with a process, 
generating a predictive model based on 
historical data about a system being 
modeled, using a validated model 
development process? to enable automatic 
transformations of variables of the data, 

aiitnmat'if* opnpratinn nf a TYTPfliptivp 

model, and automatic generation of 
performance measures of the predictive 
model on at least two independent 


In the prior approach, the model developers/users 
used their knowledge, experience and expertise to 
validate a specific model manually using a 
conventional approach and a conventional statistical 
software package, in particular, software provided 
by SAS Institute Inc., North Carolina, rather than 
using "a validated model development process." In 
the prior approach, transformations of variables of 
the data were done manually as were generation of 
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datasets of historical data, and interacting 
with the system being modeled based on 
the predictive model. 


performance measures of the predictive model on at 
least two lnoepeneno uata sets. 


Claim 7 




The method of claim 6 also including 

0f k fif k i"fitino' mpa^nirp^ of the nerfoTTriflTioe 

j^VliVl Utility HlvCVOUlvi} UJ. Uiiw L/Vl JkV^1111CU.1Vv 

of the model for the two datasets, the 
pcriuniicuit'C iiicdourcb ucmg gciicTaicu 
separately percentile by percentile 


In the prior approach, measurs of the performance 
of the model for the two datasets was performed 
manually using conventional statistical tools, expert 
estimation, and guesswork. 


Claim 8 




1 11C HlwUlUU UJ. vlCUill V dloU iJUvlUUlilg 

graphically displaying and comparing 
measures of performance for the two 
datasets. 


see notes for claim 5 


Claim 9 




The method of claim 6 also including 
persistently storing the validated model 
development process and a validated 

man a] tt\t r*r*infviitifi or T>iwr\f*ncitif*c inr Pit 
IIIUUCI IUX OUlilUUllilg piUpCllolllvo 1U1 CLl 

least one target outcome variable, the 
propensities serving as indices of a score 
for non-historical data. 


In the prior approach, there was no persistent storing 
of the validated model development process. Model 
developers/users could, however, save the model 
equation in a file on a computer along with the 
commuted nronensities 


Claim 10 




The method of claim 6 also including 
providing a user interface for assessing 
project goals against performance. 


In the prior approach, there was no user interface for 
Qcopccincr model nroiect ffoals aeainst nerformancei 
however, the model developers/users could 
manually compare mc prujcci guaia agaiuM 
performance. 


Claim 11 




The method of claim 6 also including 
providing a user interface for selecting at 
least one subset of the historical data in 
addition to a training subset. 


In the prior approach, there was no user interface for 
selecting at least one subset of the historical data in 

flHHitinn tn the trainina <nih^et* however the model 

developers/users could manually generate a subset 
Dy writing interpretive coae ior tne conventional 
statistical package. 


Claim 12 




The method of claim 6 providing a user 
interface for displaying the performance 


In the prior approach, there was no user interface for 
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of the model for at least two subsets of the 
historical data for an interval of interest. 


displaying the performance of the model for at least 
two subsets of the historical data for an interval of 
interest. However, users could manually employ 

cuiv/Uivx miiu- ucuiy ouiiwcuc ucmvuiiv, auvii cto 

Microsoft Excel, to construct a comparion chart. 


Claim 13 




The method of claim 6 enabling a user to 
choose interactivel v at least one 
transformation or interaction of variables 

IU lluprOVO U1C II1UUC1 VallUallUIl piUvCbii. 


In the prior approach, the user was not enabled to 
choose interactivelv The model develoners/users 
could, however, manually choose by writing 

interoreti vp twrle fnr the rnnvpntinnfll QtatistiPfll 

package. [Richard, is this correct?] 


Claim 14 




The method of claim 6 determining 
whether the model generalizes to the data 
other than a subsample, and, if so, 
applying a possible model to all of the 
data to generate a final model, and cross- 

vnliHafina fhe "final mnHel nsincr ranHnm 

V CUlUu.llJ.lti lilt llildl 1UUU.V1 Uoiilg ICUIUV/UI 

portions of the data. 


The prior approach did not include applying a 
possible model to all of the data to generate a final 
model or cross-validating the final model using 
random portions of the data. 


Claim 15 




The method of claim 6 providing a user 
interface that enables the user to select at 
least one validation dataset and invoke a 
model process validation method. 


The prior approach provided no user interface that 
enabled selecting a validation dataset and permitted 
invocation of a model process validation method. 
However, the model developers/users manually 
pen prated such a subset hv writing intemretive code 

ciW/11 wl Ov+Vll d OUUjvI j VY11H11& UUvl Ul VU V v vvUv 

for the conventional statistical package. 


Claim 16 




The method of claim 6 nrovidint? a user 
interface that enables the user to point and 

vllvK IU vaUoC UTopiay Ul JillOlJilaUUn 

about the model process validation. 


In the prior approach, no user interface enabled the 
user to point and click to cause display of 
information about the model process validation. 


Claim 17 




The method of claim 16 in which the 
information about the model process 
validation includes at least one of: a 
statistical report card with a link to the 
statistical report chart, a cumulative lift 
chart with a link to the cumulative lift 
chart, and a non-cumulative lift chart with 


In the prior approach, no user interface provided 
information about a model process validation that 
included at least one of: a statistical report card with 
a link to the statistical report chart, a cumulative lift 
chart with a link to the cumulative lift chart, or a 
non-cumulative lift chart. 
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of the model for at least two subsets of the 
historical data for an interval of interest. 


1 displaying the performance of the model for at least 
two subsets of the historical data for an interval of 
interest. However, users could manually employ 
another third party software package, such as 
lviiviuouii .davci, iu vAJiibiruii a L'Umpanon cnair. 


Claim 13 




The method of claim 6 enabling a user to 
choose interactively at least one 
transformation or interaction of variables 
to improve the model validation process. 


In the prior approach, the user was not enabled to 
choose interactively. The model developers/users 
could, however, manually choose by writing 
interpretive code for the conventional statistical 
package. [Richard, is this correct?] 


Claim 14 




The method of claim 6 determining 
whether the model generalizes to the data 
other than a subsample, and, if so, 
applying a possible model to all of the 
data to generate a final model, and cross- 
validating the final model using random 
portions of the data. 


The prior approach did not include applying a 
possible model to all of the data to generate a final 
model or cross-validating the final model using 
random portions of the data. 


Claim 15 




The method of claim 6 providing a user 
interface that enables the user to select at 
least one validation dataset and invoke a 
model process validation method. 


The prior approach provided no user interface that 
enabled selecting a validation dataset and permitted 
invocation of a model process validation method. 
However, the model developers/users manually 
generated such a subset by writing interpretive code 

■frvr tnp /"*r\n\/^n+ir\ri5ii ctntictifcil r\Q r*!^ - q ctp» 
IKJl U1C C-UllVClillUllal o la 11 oU^CU paviva^C. 


Claim 16 




The method of claim 6 providing a user 
interface that enables the user to point and 
click to cause display of information 
about the model process validation. 


In the prior approach, no user interface enabled the 
user to point and click to cause display of 
information about the model process validation. 


Claim 17 




The method of claim 1 6 in which the 
information about the model process 
validation includes at least one of: a 
statistical report card with a link to the 
statistical report chart, a cumulative lift 
chart with a link to the cumulative lift 
chart, and a non-cumulative lift chart with 


In the prior approach, no user interface provided 
information about a model process validation that 
included at least one of: a statistical report card with 
a link to the statistical report chart, a cumulative lift 
chart with a link to the cumulative lift chart, or a 
non-cumulative lift chart. 
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a link to the non-cumulative lift chart. 




Claim 18 




Trio *VI Q - t"rH"\.ri /"\T Ai 01 Wl 1 / m iirhii^n 

i ne metnou oi cidim i / in wnicn 
invocation of the link to the statistical 
report card causes display of the statistics 
of model process validation. 


Not present in a user interface of the prior approach. 


Claim 19 




l llC IIICUIUU Ul LialllL 1 1 111 WHICH 

invocation of the link to the cumulative 
lift chart causes display of a cumulative 
lift chart. 


Not present in a user interface of the prior approach. 


Claim 20 




i ne memoa oi claim 1 / in wnicn 
invocation of the link to the cumulative 
lift chart causes display of a non- 
cumulative lift chart. 


Not present in a user interface of the prior appoach. 


Claim 21 




The method of claim 17 in which a user is 
enauiea to cnoose inxeracuveiy at least 
one performance criterion change or 
transformation or interaction of variables 
to improve the model validation process. 


Not enabled in the nrior annroach 


Claim 22 




The method of claim 6 also including 
providing a user interface that enables the 
user to select at least one machine 
automated model development process 
applied to the entire dataset for a validated 
model process. 


Not present in the prior approach. 


Claim 23 




The method of claim 6 also including 
providing a user interface that enables the 
user to point ana ciick to cause Display or 
information about the performance of the 
validated model process applied to the 
entire set of historical data. 


In the prior approach, there was no user interface 
that enabled the user to noint and click to cause 
display of such information. 


Claim 24 




The method of claim 23 in which the 
information about the model performance 


In the prior approach, no user interface provided 
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for two independent data subsets includes 
at least one of the following: a statistical 
report card with a link to the statistical 

ICpUll vllttll, a L/UlliUlclll VC llli. V/llctll W1UX a 

link to the cumulative lift chart, a non- 
cumulative lift chart with a link to the 
non-cumulative lift chart. 


information about a model performance for two 
independent data subsets that included at least one 
of: a statistical report card with a link to the 

QtntiQtiffil rpnrvrt pVinrt n fiTmiilfltivp liir pnart witn a 

link to the cumulative lift chart, or a non-cumulative 
lilt cnart. 


Claim 25 




1 llv- IJllwlliV/U ul VI CUi.il HI W IJULVU UJlv 

invocation of the link to the statistical 
repon cara causes uispiay oi me siausucs 
of model process validation. 


Not present in the user interface of the prior 
armroach 


Claim 26 




i nc mcuiou ui ciaiiii m wiuvn mc 
invocation of the link to the cumulative 
lift chart causes display of a cumulative 
lift chart. 


Not present in the user interface of the prior 

flTvrvrnaf4i 


Claim 27 




TKo to f*tVi r\ri r\T fOaim OA. in ix/niPn trip 
1 IlC II1CU1UU Ui Clallll £H ill Wlllt/ll LUC 

invocation of the link to the cumulative 
lift chart causes display of a non- 
cumulative lift chart. 


Not present in the user interface of the prior 
annroach 


Claim 28 




The method of claim 6 also including 
storing the final model and the model 
process validation results persistently. 


In the prior approach, the model developers/users 
manuallv generated interpretive code for the 
conventional statistical package to persistly store in 
computer file storage the final model and numeric 
model validation results. 
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Dr. Mansfield has more than 15 years of successful, progressive experience in all 
phases of mission-critical software product and systems development. He is the author 
or co-author of a number of software patents. His role at Fortelligent is to design and 
develop Fortelligenfs OASYS Analytic Platform™ and to provide technical oversight of 
Fortelligent's systems, software and processes to ensure optimization for business 
advantage. 

Dr. Mansfield earned his Ph.D. from Harvard University and a Master's from Yale 
University in Mathematics. He was a Visiting Scholar at the Johns Hopkins School of 
Medicine as well. 
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Steve Pinto has been at the forefront of predictive analytics for over 30 years. Over 
that time he has formulated many of the modern-day methodologies used in database 
marketing and modeling. He is responsible for Product Development, Account 
Management, and client deliverables at Fortelligent. His prior experience includes: 

At Mercer Management Consulting, a global strategy firm, Steve was a Vice President 
and a lead developer of the Customer Relationship Management practice. He was 
responsible for developing the worldwide practice for CRM at Mercer. The practice 
develops analytical CRM program on behalf of diversified Global 2,000 clients and 
manages those programs on an ongoing basis to help generate increasing shareholder 
value. 

Prior to his work at Mercer Management Consulting, Steve was the Managing Director 
of the Analytic Consulting Group at Epsilon, a former division of American Express. 
Steve was responsible for creating and deploying analytical CRM programs on behalf 
of the largest financial institutions in the country. 

Before Epsilon, Steve Co-Founded Applied Research of Cambridge, a worldwide 
market research firm later acquired by Cap Gemini, a global strategy consulting firm. 

Steve is a widely published author on the subject of predictive analytics and is on 
several editorial boards including the Journal of Database Marketing, 
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Fortelligent's Scientific Advisor is the world-renown statistician, Professor Donald 
Rubin. His role at Fortelligent has been to contribute to the development of 
Fortelligent's Platform by designing unique algorithms and methodologies that enable 
Fortelligent to maintain an ongoing competitive edge. 

Professor Rubin is a world-leading thinker in Statistical Science. In 2002 he was one of 
the most quoted scholars in the field of statistics. Currently he is Chairman of the 
Statistics Department at Harvard University, a position he has held for 20 years. He is 
the John L. Loeb Professor of Statistics and key PhD advisor to Candidates for degrees 
in Statistics at Harvard as well. 

Prior roles held at Harvard University include: 

• Chairman, Department of Statistics Senior Search Committee 

• Director of Graduate Studies, Department of Statistics 
Faculty Staff, Program on Educational Policy and Governance 



Professor Rubin is a widely known force in the statistical community, having published 
over 300 articles and book on various statistical-related topics. 



